Systems and methods for implementing an intelligent application program interface for an intelligent optimization platform

ABSTRACT

Systems and methods for implementing an application programming interface (API) that controls operations of a machine learning tuning service for tuning a machine learning model for improved accuracy and computational performance includes an API that is in control communication the tuning service that: executes a first API call function that includes an optimization work request that sets tuning parameters for tuning hyperparameters of a machine learning model; and initializes an operation of distinct tuning worker instances of the service that each execute distinct tuning tasks for tuning the hyperparameters; executes a second API call function that identifies raw values for the hyperparameters; and generates suggestions comprising proposed hyperparameter values selected from the plurality of raw values for each of the hyperparameters; and executes a third API call function that returns performance metrics relating to a real-world performance of the subscriber machine learning model executed with the proposed hyperparameter values.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is an application is a continuation of U.S. application Ser. No.16/450,891, filed 24 Jun. 2019, which is a continuation of U.S. Pat. No.10,379,913, filed 20 Mar. 2019, which is a continuation of U.S. Pat. No.10,282,237, filed 29 Oct. 2018, which claims the benefit of U.S.Provisional Application No. 62/578,886, filed 30 Oct. 2017, which areall incorporated herein in their entireties by this reference.

BACKGROUND

Modern machine learning capabilities are rapidly changing and improvinghow some of the most complex and data-intensive computing problems aresolved. A performance of a machine learning model is governed mainly inthe manner(s) in which the machine learning model is trained and basedon the hyperparameters of the machine learning model set prior to thetraining of the model. As referenced in passing the hyperparameters ofthe machine learning models are parameters whose values are set prior tothe commencement of the machine learning process rather than derived bythe machine learning model during training. Example include the numberof trees in a random forest or the number of hidden layers in a deepneural net. Adjusting the values of the hyperparameters of a machinelearning model by any amount typically results in a large impact on aperformance of the machine learning model.

However, many existing machine learning models are not implemented withoptimal hyperparameters well-suited for achieving the best predictiveperformances. Rather, the many existing machine learning models areimplemented with default hyperparameters that have not been optimizedfor a specific computing problem for which the machine learning modelsare being used.

Additionally, any existing system that enables optimization ofhyperparameters of a machine learning model typically includes anextremely complex interface that may require significant codingcapabilities and comprehension of the underlying software and hardwarecomponents of the system. Thus, making it difficult to efficiently andeffectively perform optimizations and subsequent improvements of themachine learning models.

Thus, there is a need in the machine learning field to create animproved optimization platform to test and improve machine learningmodels (e.g., in-product machine learning models) and an associatedApplication Program Interface that enables developers to efficiently andeffectively interact with a robust system implementing the evaluationframework. The embodiments of the present application described hereinprovide technical solutions that address, at least, the need describedabove, as well as the technical deficiencies of the state of the artdescribed throughout the present application.

SUMMARY OF THE INVENTION

In one embodiment, a system implementing an application programminginterface (API) that controls one or more operations of a machinelearning tuning service for tuning a machine learning model for improvedaccuracy and computational performance, the system includes a machinelearning tuning service that is hosted on a distributed networkedsystem; an intelligent application programming interface (API) that isin operable control communication of the machine learning tuning servicethat: executes a first API call function that: includes an optimizationwork request that sets a plurality of tuning parameters for tuning, bythe machine learning tuning service, hyperparameters of a subscribermachine learning model, wherein the subscriber machine learning modelincludes a machine learning model that is external to the machinelearning tuning service and implemented by a subscriber to the machinelearning tuning service; initializes an operation of each of a pluralityof distinct tuning worker instances of the machine learning tuningservice that execute distinct tuning tasks for tuning thehyperparameters of the subscriber machine learning model according tothe plurality of tuning parameters; executes a second API call functionthat: identifies a plurality of raw values for the hyperparametersgenerated by the machine learning tuning service; generates one or moresuggestions comprising proposed hyperparameter values selected from theplurality of raw values for each of the hyperparameters of thesubscriber machine learning model; and executes a third API callfunction that returns to the machine learning tuning service performancemetrics relating to a real-world performance of the subscriber machinelearning model executed with the proposed hyperparameter values for thehyperparameters of the subscriber machine learning model.

In one embodiment, the hyperparameters comprise parameters of thesubscriber machine learning model having values set before a training orthe learning process applied to the subscriber machine learning modeland are not derived during the training or the learning processing.

In one embodiment, executing the second API call function includes:instantiating a surrogate machine learning model for the subscribermachine learning model; and testing a performance of the surrogatemachine learning model based on input of the plurality of raw values foreach of the hyperparameters of the subscriber machine learning model.

In one embodiment, the surrogate machine learning model comprises ahypothesis of or a proxy for the subscriber machine learning model whena structure of the subscriber machine learning model is not known to themachine learning tuning service.

In one embodiment, instantiating the surrogate machine learning modelincludes estimating an algorithmic structure of the surrogate modelbased on the hyperparameters of the tuning work request and theplurality of raw values for each of the hyperparameters of thesubscriber machine learning model.

In one embodiment, executing the second API call function furtherincludes: identifying a subset of the plurality of raw values for eachof the hyperparameters based on results of the testing with thesurrogate machine learning model, wherein the proposed hyperparametervalues of the one or more suggestions comprise the subset of theplurality of raw values.

In one embodiment, identifying the subset of the plurality of raw valuesfor each of the hyperparameters includes identifying distinct raw valuesfrom the plurality of raw values that satisfy or exceed a performancethreshold.

In one embodiment, the machine learning tuning service includes: acluster of distinct machine learning tuning sources that operate togenerate the plurality of raw values for the hyperparameters for thesubscriber machine learning model, wherein each of the plurality oftuning worker machine selectively operates one distinct machine learningtuning source of the cluster of distinct machine learning tuningsources.

In one embodiment, the intelligent application programming interface(API) further: executes an intervening API call function to generate theone or more suggestions ahead of an estimated or a scheduled completionof the optimization work request, wherein in response to executing theintervening API call function, instantiating a surrogate machinelearning model in lieu of the subscriber machine learning model; andtesting a performance of the surrogate machine learning model based oninput of the plurality of raw values for each of the hyperparameters ofthe subscriber machine learning model.

In one embodiment, in response to the intervening API call function,automatically selecting by one of the tuning worker machines one of theplurality of distinct machine learning tuning sources to generate asurrogate model for testing a performance of the plurality of raw valuesfor the hyperparameters.

In one embodiment, executing the first API function call sets a mode ofoperation of the machine learning tuning service which is selected froma plurality of distinct modes of operation based on at least one tuningparameter provided with the optimization work request.

In one embodiment, executing the first API function call sets one ormore switching conditions that, when satisfied, switches a mode ofoperation of the machine learning tuning service from a first mode ofoperation to a second mode of operation, wherein the machine tuningservice optimizes the hyperparameters differently in the second mode ofoperation than in the first mode of operation.

In one embodiment, the operation of each of the plurality of distincttuning worker instances is asynchronous.

In one embodiment, the machine learning tuning service executes aplurality of first API function calls for a plurality of optimizationwork requests for the subscriber to the machine learning tuning service,responsive to the execution of the plurality of first API functioncalls, the machine learning tuning service instantiates distinctclusters of tuning worker instances that each operate independently tooptimize hyperparameters one of the plurality of optimization workrequests.

In one embodiment, an online method implementing an applicationprogramming interface (API) that controls one or more operations of amachine learning tuning service for tuning a machine learning model forimproved accuracy and computational performance, the online methodincludes implementing a remote machine learning tuning service that ishosted on a distributed networked system; implementing an applicationprogramming interface (API) that is in operable control communication ofthe machine learning tuning service that: executes a first API callfunction that: includes an optimization work request that sets aplurality of tuning parameters for tuning, by the machine learningtuning service, hyperparameters of a subscriber machine learning model,wherein the subscriber machine learning model includes a machinelearning model that is external to the machine learning tuning serviceand implemented by a subscriber to the machine learning tuning service;initializes an asynchronous operation of each of a plurality of distincttuning worker instances of the machine learning tuning service thatexecute distinct tuning tasks for tuning the hyperparameters of thesubscriber machine learning model according to the plurality of tuningparameters; executes a second API call function that: identifies aplurality of raw values for the hyperparameters generated by the machinelearning tuning service; generates one or more suggestions comprisingproposed hyperparameter values selected from the plurality of raw valuesfor each of the hyperparameters of the subscriber machine learningmodel; and executes a third API call function that returns to themachine learning tuning service performance metrics relating to areal-world performance of the subscriber machine learning model executedwith the proposed hyperparameter values for the hyperparameters of thesubscriber machine learning model.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a schematic representation of a system in accordancewith one or more embodiments of the present application;

FIG. 2 illustrates a method for implementing an intelligent API inaccordance with one or more embodiments of the present application;

FIG. 3 illustrates a schematic representation of a system forimplementing an intelligent API in accordance with one or moreembodiments of the present application; and

FIG. 4 illustrates schematic representation of a mixed system andprocess flow for implementing an intelligent optimization platform inaccordance with one or more embodiments of the present application.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the presentapplication are not intended to limit the inventions to these preferredembodiments, but rather to enable any person skilled in the art to makeand use these inventions.

Overview

As discussed above, existing machine learning models tend to lack inpredictive performance as well as speed in computation due to a lack ofoptimal hyperparameters used in the machine learning models duringtraining. The lack of optimized hyperparameters well-suited to anunderlying computing problem or the like adversely affect thecomputational capabilities of the machine learning model, in that, theresulting predictions or solutions of the model may not be accurate andthe speed of computation of the machine learning model may be slowbecause the un-optimized or poorly optimized hyperparameters of themodel may result in an increased workload (e.g., increased requiredcomputer processing, increased required data storage, etc.) to thecomputing system implementing the model and thereby resulting in manyefficiencies therein.

Additionally, even in the circumstance that an attempt is made tooptimize some of the hyperparameters of a machine learning model, theattempt to optimize the hyperparameters may fail due to many commonoptimization errors including: using inappropriate metrics andassumptions to test hyperparameters of an associated machine learningmodel or the like; overfitting a machine learning model during trainingthat often results in a poor fit of the model to out of sample data orunseen data; using too few hyperparameters and failing to optimize allhyperparameters of a machine learning model; unskilled or improperhand-tuning, which is a highly inefficient optimization strategy atwhich humans are poor at performing high dimensional, non-convexoptimization; grid searching over a space of possible hyperparameterswhich may grow the number of times a machine learning model must beevaluated due to an increase in dimensionality (i.e., increasedhyperparameters); random searching which uses no intelligence in theoptimization method and may result in unnecessarily high variance.

Accordingly, unintelligent optimization attempts of hyperparameters (orother model parameters) may result in high computational costs (e.g.,high computer processing resources expenditures, etc.).

The embodiments of the present application, however, provide anintelligent optimization platform that functions to optimizehyperparameters and/or parameters of any type of model withsignificantly fewer evaluation thereby saving computational resourceswhile greatly improving an overall performance of a model. Inembodiments of the present application, the intelligent optimizationplatform includes an ensemble of parameter optimization models, whichmay include a combination of several distinct machine learning modelsand Bayesian optimization algorithms that may work in selectivecombinations to expediently tune hyperparameters or various parametersof complex external systems, simulations, and models.

Further, the embodiments of the present application include an intuitiveand simplified Application Programming Interface (API) that enablesusers and/or developers to easily configure a work request, such as ahyperparameter optimization work request. A hyperparameter optimizationwork request as referred to herein generally relates to a request tooptimize one or more hyperparameters of a model. The hyperparameteroptimization work request may include an identification of thehyperparameters a user desires to optimize together with constraints orparameters required for experimenting or performing optimization trialsusing the system and/or methods described herein. The optimization workrequest may generally be generated using an API of the system 100, asdescribed below. In a preferred embodiment, the optimization workrequest functions to trigger an operation of the intelligentoptimization platform performing computations using the hyperparametersof the optimization work request. Additionally, in embodiments of thepresent application, using a limited number of simplified API calls, itis possible to integrate the sophisticated ensemble of Bayesianoptimization techniques of the intelligent optimization platform toaugment an existing machine learning pipeline.

Collaboratively, the intelligent optimization platform preferablyfunctions to improve the computational capabilities of a machinelearning model, such that the machine learning model performs at highlevels of accuracy and further, computes predictions, suggestions, andother outcomes faster (e.g., up to one hundred times faster or moreimprovement in machine learning models, etc.) than un-optimized orpoorly optimized machine learning models or other models. This, in turn,improves the functionality and operational speed and efficiency of theunderlying computing system executing the machine learning model orother model.

1. System for Implementing an Intelligent API

As shown in FIG. 1, a system 100 an intelligent application programinterface (API) 105, an intelligent model optimization platform 110, aplurality of queue working machines 120, a platform database 130, ashared work queue 135, and an ensemble of optimization models 140.

The system 100 preferably implements an intelligent model optimizationplatform 110 including an ensemble of Bayesian optimization processesand machine learning techniques that functions to automate anoptimization of features and/or hyperparameters of a model, architectureof a model, and hyperparameters of a model using an ensemble of Bayesianoptimization techniques, as described in U.S. patent application Ser.No. 15/977,168, U.S. Patent Application No. 62/587,827, and U.S. PatentApplication No. 62/721,718, which are incorporated herein by thisreference in their entireties.

The system 100 functions to implement an intelligent Application ProgramInterface (API) 105 for interacting and implementing complexoptimization trials via the remote intelligent optimization platform110. The API 105 may be specifically designed to include a limitednumber of API endpoints that reduce of complexity in creating anoptimization work request, implementing optimization trials using thework request data, obtaining suggestions and/or results of theoptimization trials, and potentially implementing an optimizationfeedback loop until a suitable optimization of an objective function ofthe work request is achieved in a minimal amount of time. Theoptimization work request, as referred to herein, generally relates toan API request that includes one or more parameters that a user isseeking to optimize and one or more constraints that the user desiresfor the optimization trials performed by the intelligent optimizationplatform 110.

In a preferred embodiment, the API 105 comprises a RepresentationalState Transfer (ReST) API that relies mainly on a stateless,client-server, cacheable communications protocol and in many cases, theRest API uses the HTTP protocol in connecting and interacting withsoftware applications over the web and cloud (distributed networksystems) services efficiently.

The API 105 may additionally be configured with logic that enables theAPI 105 to intelligently parse optimization work request data and/oraugment the optimization work request data with metadata prior topassing the optimization work request to the shared work queue 135 ofthe intelligent optimization platform 110. As shown in FIG. 4, a mixedsystem and process flow is provided that illustrates an exampleinteractions between the API 105 and one or more components of theintelligent optimization platform 110.

The intelligent optimization platform 110 includes the plurality ofqueue worker machines 120 (which may also be referred to herein asoptimization worker machines), the platform data 130, the shared workqueue 135 and the ensemble of optimization models 140. The intelligentoptimization platform 110 generally functions to interact with the APIserver implementing the API 105 to receive API requests for implementingnew optimization work requests and returning responses or suggestions tothe API 105. Using the plurality of intelligent queue worker machines120, the intelligent optimization platform 110 functions toasynchronously execute a plurality of optimization work requests inreal-time and in parallel. This asynchronous execution and parallelprocesses of the intelligent optimization system 110 provides speed incomputing efficiencies in the exploration and exploitation processes(generally, optimization) of features, hyperparameters, models andsystem architectures.

As shown by way of example in FIG. 3, the system enables a user toimplement and/or interact with the API 105 in multiple ways includingvia an API client application and/or via API web browser implementedover the web.

The intelligent optimization platform 110 may be implemented using acombination of computing servers. Preferably, the intelligentoptimization platform is implemented via a distributed networkedcomputing system, such as cloud computing systems, that allows the manyprocesses implemented by the intelligent optimization platform 110 to beimplemented in parallel and among disparate computers thereby, in someembodiments, mitigating the possibility of failure or bottlenecking inthe optimization pipeline of the intelligent optimization platform 110.Accordingly, the intelligent optimization platform 110 may beimplemented as a remote web service accessible by multiple clients overthe Internet, the Web, or any suitable communication network (e.g., aglobal area network, a wide area network, a local area network, etc.)that may function to place disparate computing resources in operableconnection and communication.

The plurality of intelligent queue worker machines 120 preferably relateto services operating on the intelligent optimization platform 110 thatexecutes code asynchronously with respect to other services or queueworking machines of the platform 110. In some embodiments, each of theplurality of intelligent queue worker machines 120 functions toselectively trigger one or more optimization requests to one or moreoptimization engines of the ensemble of optimization engines 140. And,once the work on the optimization request is completed by the selectedoptimization engine(s), the queue working machine returns the responsesor results to the platform database 130.

The plurality of intelligent queue worker machines 120 may bespecifically configured with logic that enables each of the machines 120to make dynamic and intelligent decisions in the selections of anensemble component of the plurality of ensemble of optimization models140. That is, each of the plurality of intelligent queue worker machinesmay function to selectively choose one or more optimization models ofthe ensemble 140 to execute one or more portions of an optimization workrequest.

The ensemble of optimization models 140 may include any number of modelsincluding, for example: a Low-Discrepancy sequence model, a MetricOptimization Engine (MOE) model (and variants thereof; e.g., MOE withone-hot encoding), a Tree-structured Parzen Estimators (TPE) model andvariants thereof, a Latin Hypercube model, a Swarm model, and the like.Each of these models of the example ensemble of optimization models mayfunction to encode categorical parameters differently from other membermodels of the ensemble and may include some interdependencies thatrequire combinations of the models to work together. Each of thesemodels may be individually selectable or selectable in combination bythe intelligent worker queue machines 120.

The ensemble of optimization models 140 preferably includes a pluralityof disparate optimization models that operate to optimize parameters,features, models, system architectures and the like using varyingoptimization algorithms. In a preferred embodiment, the ensemble ofoptimization models 140 define a core optimization engine of theintelligent optimization platform 110. The features and the parametersof the core optimization engine comprising the ensemble of optimizationmodels 140 may also be optimized continually by one or more of theintelligent queue worker machines 120 (e.g., using Hyperopt, etc.).

The ensemble of optimization models 140 may include any number of modelsincluding, for example: a Low-Discrepancy sequence model, a MetricOptimization Engine (MOE) model (and variants thereof; e.g., MOE withone-hot encoding), a Tree-structured Parzen Estimators (TPE) model andvariants thereof, a Latin Hypercube model, a Swarm model, and the like.Each of these models of the example ensemble of optimization models mayfunction to encode categorical parameters differently from other membermodels of the ensemble and may include some interdependencies thatrequire combinations of the models to work together. Each of thesemodels may be individually selectable or selectable in combination by orusing the intelligent worker queue machines 120.

In a preferred embodiment, the plurality of intelligent queue workingmachines 120 may be implemented on a separate computing server than theAPI 105. In this way, long-running asynchronous processes do notadversely affect (e.g., slow down) a performance of an API computingserver and mainly, a capacity of the API computing server to service APIrequests.

Additionally, the plurality of intelligent queue worker machines 120include multiple, distinct intelligent queue worker machines 120 thatcoordinate optimization work request from the shared work queue 135received via the API 105 with the ensemble of optimization models 1400.

A first example intelligent queue working machine may function toimplement Modelfit or Hyperopt that typically functions to tune one ormore of the hyperparameters of the optimization models of the ensembleconcurrently with the processing of the optimization work requestsreceived via the API 105. In one implementation, Modelfit or Hyperoptmay be used to tune hyperparameters of one of the optimization models ofthe ensemble 140. After receiving a set of observations based on thesuggestions for the set of parameters, the first queue worker machinemay implement Modelfit or Hyperopt to model fit the hyperparameters ofthe selected optimization models in order to generate improved and newvalues for the set of hyperparameters via Nextpoints or the like. Insome embodiments, the first queue worker machine may function tooptimize the hyperparameters of the selected optimization models basedon a return of a set of observations. However, it shall be noted thatobservations may be generated in any suitable manner including in theabsence of suggestions by the system 100.

A second example intelligent queue working machine may function toimplement Nextpoints that typically functions to generate or suggestnew, optimized values for the parameters of the optimization workrequest. Accordingly, such intelligent queue working machine mayfunction to select one or more of the optimization models of theensemble 140, such as one or more machine learning models, forgenerating the new, optimized parameter values.

A third example intelligent queue working machine may function toimplement an Importance algorithm that typically functions to judge ordetermine an importance of the hyperparameters submitted with theoptimization work request (e.g., hyperparameters of an external model).This example intelligent queue worker machine may additionally functionto analyze and determine an importance of features, hyperparameters, andarchitectures of the optimization models with respect to a givenoptimization work request; meaning the identified importancehyperparameters, features, or the like may have a significant impact onan account of a suggestion or generated parameter values. Accordingly,the intelligent queue worker machine of such example may function torecognize different hyperparameters and/or features of an optimizationmodel as being important and non-important based on the optimizationwork request data (e.g., based on the hyperparameters to be optimized).Thus, the intelligent queue worker machine may function to assign orattribute distinct importance values to the hyperparameters and/orfeatures of the optimization models so that these hyperparameters andthe like may be ranked and considered with greater weight in acorrelated process, such as re-tuning via Hyperopt or the like.

It shall be noted that the plurality of intelligent optimization workermachines 120 may not be limited to the above-noted examples, but ratheris an extensible group of intelligent machines that may be modified toinclude additional and/or different intelligent worker machines that maybe configured or programmed to perform any suitable optimization task ofhyperparameters of machine learning models and/or related tasks to theprocesses and methods described herein.

The platform database 130 functions to collect and stores any or allvalues generated by the system 100 including values generated whenexecuting an optimization work request by the intelligent optimizationplatform 110. Specifically, each of the plurality of intelligent queueworker machines may function to store within the platform database 130optimized hyperparameter values, optimized parameter values of anoptimization work request, suggestions, surrogate models, partialinformation responses, and the like. The API 105 may be operablecommunication with the platform database 130 via a communication networkand may function to pull suggestions and/or response data via an APIcall or request.

The machine learning models, optimization models, and/or the ensemble ofmachine learning models may employ any suitable optimization algorithmsand/or machine learning including one or more of: supervised learning(e.g., using logistic regression, using back propagation neuralnetworks, using random forests, decision trees, etc.), unsupervisedlearning (e.g., using an Apriori algorithm, using K-means clustering),semi-supervised learning, reinforcement learning (e.g., using aQ-learning algorithm, using temporal difference learning), and any othersuitable learning style. Each module of the plurality can implement anyone or more of: a regression algorithm (e.g., ordinary least squares,logistic regression, stepwise regression, multivariate adaptiveregression splines, locally estimated scatterplot smoothing, etc.), aninstance-based method (e.g., k-nearest neighbor, learning vectorquantization, self-organizing map, etc.), a regularization method (e.g.,ridge regression, least absolute shrinkage and selection operator,elastic net, etc.), a decision tree learning method (e.g.,classification and regression tree, iterative dichotomiser 3, C4.5,chi-squared automatic interaction detection, decision stump, randomforest, multivariate adaptive regression splines, gradient boostingmachines, etc.), a Bayesian method (e.g., naïve Bayes, averagedone-dependence estimators, Bayesian belief network, etc.), a kernelmethod (e.g., a support vector machine, a radial basis function, alinear discriminate analysis, etc.), a clustering method (e.g., k-meansclustering, expectation maximization, etc.), an associated rule learningalgorithm (e.g., an Apriori algorithm, an Eclat algorithm, etc.), anartificial neural network model (e.g., a Perceptron method, aback-propagation method, a Hopfield network method, a self-organizingmap method, a learning vector quantization method, etc.), a deeplearning algorithm (e.g., a restricted Boltzmann machine, a deep beliefnetwork method, a convolution network method, a stacked auto-encodermethod, etc.), a dimensionality reduction method (e.g., principalcomponent analysis, partial lest squares regression, Sammon mapping,multidimensional scaling, projection pursuit, etc.), an ensemble method(e.g., boosting, boostrapped aggregation, AdaBoost, stackedgeneralization, gradient boosting machine method, random forest method,etc.), and any suitable form of machine learning algorithm. Eachprocessing portion of the system 100 can additionally or alternativelyleverage: a probabilistic module, heuristic module, deterministicmodule, or any other suitable module leveraging any other suitablecomputation method, machine learning method or combination thereof.However, any suitable machine learning approach can otherwise beincorporated in the system 100. Further, any suitable model (e.g.,machine learning, non-machine learning, etc.) can be used inimplementing the intelligent optimization platform 110 and/or othercomponents of the system 100.

The system 100 may additionally include a surrogate model generator(implemented via one or more of the optimization models 140) that may beimplemented by the intelligent optimization platform 110. Specifically,when an API request is received by the system 100 that requests a statusor suggestions of a partially completed (or incomplete) optimizationwork request, the intelligent optimization platform 110 may function toidentify candidate data points and other data (including suggestedparameter values and optimized hyperparameters values) generated by eachof the plurality of intelligent queue worker machines 120 for respondingto the partially completed optimization work request and further, maytrigger one or more of the optimization models of the ensemble ofoptimization models to generate a surrogate (or proxy) model that can beused to test the uncertainty and/or the likelihood that a candidate datapoint would perform well in an external model. In one example, thesystem 100 may function to obtain hyperparameter values of a most recentjob of a first intelligent queue worker machine implementing Hyperoptand cause one of the optimization models 140, such as MOE, to generatethe surrogate model using the hyperparameter values to test how well thecandidate parameter value data points may perform.

The system 100 may also implement a ranking system 155 that functions torank the suggestions for a given optimization work request (or acrossmultiple optimization work requests for a given user) such that thesuggestions having parameter values most likely to perform the best canbe passed or pulled via the API 105. The ranking system 155 may beimplemented in any suitable manner including by the one or moreoptimization algorithms of the ensemble 140 that generated thesuggestions. For instance, if MOE is used to generate a plurality ofsuggestions for responding to an optimization work request, the system100 may function to use MOE to implement the ranking system 155.

It shall be noted that the sub-systems and components of the system 100may be connected or placed in operable communication using any suitablenetwork and any suitable manner. For instance, the components of thesystem 100 may be connected directly or indirectly over a network. Thenetwork may include any public (e.g., the Internet) or private network(e.g., intranet), a virtual private network, a wireless local areanetwork, a local area network, a wide area network, a wireless wide areanetwork, a global area network, a cellular network, any combination ofthe aforementioned and the like.

2. Method for Implementing an Intelligent API

As shown in FIG. 2, a method 200 for implementing an intelligentapplication program interface (API) that enables efficient and intuitiveinteraction with a remote intelligent optimization platform includesaccessing the intelligent API S205, implementing a first API function(e.g., a first API call) to create an optimization work request S210,implementing a second API function (e.g., a second API call) to accesssuggestions from the intelligent optimization platform for the tunableparameters of the optimization work request 220, implementing a thirdAPI function (e.g., a third API call) to create an observation objectvia the intelligent API S230, and implementing an optimization feedbackloop based on receipt of observations data via the intelligent API S240.

S205, which includes accessing the intelligent API S205, functions toenable a remote user or user system to establish an API account and APIaccount access via one or more modes.

In a first implementation, S205 enables a user or user system to enrollvia a web browser that is operably connected to the systems implementingthe remote intelligent optimization platform. Using the web browser, auser and/or user's system may function to establish API accountcredentials and a user profile.

In a second implementation, S205 enables a user or user system todownload an API client application that once download and implemented bya user enables the user to enroll for an API account similar to thefirst implementation.

In response to a successful enrollment, S205 may function to provide ortransmit an API key to the user. The API key provided by S205 mayfunction to enable the user to interact with the API account as well asimplement one or more functionalities of the intelligent API and also,generate API requests and access one or more web resources via theintelligent API.

S210, which includes implementing or executing a first API function tocreate an optimization work request, functions to enable a user to inputdetails of a desired optimization work request to be processed by theintelligent optimization platform. That is, the first API functionenables a user to define an experiment or optimization trials includingidentifying variables, such as hyperparameters, constants (e.g., bounds,etc.), performance thresholds, and the like (tuning parameters) that maybe used by the method 200 to govern a manner in which an intelligentoptimization platforms an execution of the optimization work request.

Accordingly, in a preferred embodiment, S210 may function to use thefirst API function to create or build an optimization work request thatsets a plurality of tuning parameters for tuning or optimizinghyperparameters of a machine learning model (e.g., external model) of asubscriber to the one or more services of the intelligent optimizationplatform. In such preferred embodiment, S210 may additionally oralternatively function to initialize one or more optimization operationsor functions of the intelligent optimization platform including, but notlimited to, initializing an operation of each of a plurality of distincttuning worker instances of the platform that execute distinct tuningtasks for tuning the hyperparameters of the machine learning modelaccording to the plurality of tuning parameters. Preferably, thehyperparameters comprise parameters of an external machine learningmodel having values set before a training or the learning processapplied to the external machine learning model and are not derivedduring the training or the learning processes involving the externalmachine learning model.

In a first implementation, if a user implements the first API functionto create an optimization work request via an API web browser or thelike, the intelligent API may function to provide an interactiveillustration via the API web browser that demonstrates one or moremethodologies for generating an optimization work request via theintelligent API.

In a second implementation, if the user implements the first APIfunction to create an optimization work request via the API clientapplication, the intelligent API enables a user to provide input orexperiment details for creating the optimization work request directlywith API code. Preferably, the API code of the system may include acustom API code that is specifically designed for the intelligent APIand for controlling one or more operational aspects of the intelligentoptimization platform. Thus, the API code terms may include a pluralityof predefined terms and/or characters that may be comprehendible by theintelligent API and consequently, create executable controls foroperating the intelligent optimization platform.

In a preferred embodiment, creating the input details for generating anoptimization work request via the intelligent API includes descriptiveand parameter inputs of the optimization work request. For instance, oneor more required input fields may include: providing name input foridentifying the optimization work request via the intelligent API;providing the tunable hyperparameters for the optimization work requestthat the user desires for the intelligent optimization platform tooptimize to achieve a desired metric of a machine learning model or thelike; providing bounds input (e.g., max/min) that function to constrainthe possible values that the tunable parameters may occupy duringoptimization; precision input via that defines a level of precisionrequired for the parameter values (e.g., int or integer; double orinclude decimal values, etc.); providing type_categorical input orcategorical values (or list) that the intelligent optimization platformcan select from during an optimization session or trial; providingmetrics input indicating the metrics (e.g., performance metricsincluding accuracy, computational speed, etc.) to be measured; anobservation budget input indicating an amount of suggestions (e.g.,distinctly generated values for the hyperparameters) the user desiresthe intelligent optimization platform to generate; and metadata inputthat enables tracking of one or more aspects of the optimization workrequest.

In one implementation, the observation budget input provided via theintelligent API may function to configure or set multiple operationmodes of the intelligent optimization platform. That is, S210 mayfunction to execute the first API function call to set a mode ofoperating of the intelligent optimization platform, which is selectedfrom a plurality of distinct modes of operation based on at least onetuning parameter (e.g., budget) provided with the optimization workrequest. The multiple operation modes may include a first explorationmode, a second combined exploration/exploitation mode, and a thirdexploitation mode.

Additionally, or alternatively, upon receiving the observation budgetindicating a number of suggestions that a user desires the intelligentoptimization platform to generate based on one or more optimization workrequests, S220 may function to configure mode triggering thresholds thatmay be used by the intelligent optimization platform to selectivelyswitch an operation mode of the platform. That is, S210 may function toexecute the first API function call to set one or more mode switchingconditions that, when satisfied by one or more metrics of theintelligent optimization platform during an execution of theoptimization work request, automatically switches a mode of operation ofthe intelligent optimization platform from a first mode of operation toa second mode of operation. In such embodiments, the intelligentoptimization platform functions to tune or optimize the hyperparametersdifferently in the second mode of operation than in the first mode ofoperation. For instance, based on an optimization budget, S220 mayfunction to set a first mode switching triggering threshold thattransitions an optimization of the hyperparameter values from a firstexploration mode to a second combined exploration/exploitation mode.Additionally, in such instance, S220 may function to set a second modeswitching threshold that transitions the intelligent optimizationplatform from the second combined exploration/exploitation mode to athird exploitation mode of the intelligent optimization platform.

Specifically, for each operation mode of the intelligent optimizationplatform, only a predetermined set of optimization models of theensemble of optimization models may be implemented or requested by aplurality of optimization worker machines of the platform. That is, eachmode limits the optimization resources available to the plurality ofoptimization worker machines. In each mode, the predetermined set ofoptimization models is preferably different, such that each set ofpredetermined optimization models available under each operation modefunctions to produce different results given a same set ofhyperparameters and/or values to optimize.

Accordingly, the mode triggering thresholds for each operation mode ispreferably set by S220 using the observation budget prior to orcontemporaneously with a processing of the optimization work request(s)by the intelligent optimization platform. As an example, using thefunction API function, a user may set an observation budget value oftwenty (20) suggestions. The intelligent API may function to set at anexploration mode threshold to zero; meaning at the initialization of theoptimization processes, the intelligent optimization platform should beconfigured to use a first predetermined set of optimization models thatfunction to explore preferably in a wide range of values, potentialvalues for the hyperparameters of an optimization work request. Theintelligent API may additionally function to set anexploration/exploitation mode threshold to ten (10), such that once tensuggestions have been created in the exploration mode, the intelligentoptimization platform automatically switches to and configures theexploration/exploitation mode which requires the intelligentoptimization platform to use a second predetermined set of optimizationmodels (e.g., tuning sources) for performing a combination ofexploration with hyperparameter values and exploitation of any suggestedhyperparameter values. Lastly, in such example, the intelligent API mayfunction to set an exploitation mode threshold to fifteen (15), suchthat once a total of fifteen suggestions for the optimization workrequest have been suggested or generated in the prior exploration modeand the exploration/exploitation mode, the intelligent optimizationplatform switches to the exploitation mode which requires theintelligent optimization platform to use a third predetermined set ofoptimization models for performing only an exploitation of any suggestedhyperparameter values.

Accordingly, configuring the intelligent optimization platform to switchoperation/optimization modes enables a balanced approach to optimizinghyperparameter values of a model according to an observation budget ofthe optimization work request thereby avoiding over exploration orexploitation and under exploration and exploitation.

Once a user successfully executes the first API function to create anoptimization work request, the intelligent API may trigger a start ofthe optimization process by passing the optimization work request to theintelligent optimization platform.

It shall be noted that the intelligent optimization platform mayfunction to execute a plurality of first API function calls for aplurality of optimization work requests for a given subscriber to theservices of the intelligent optimization platform. Additionally, oralternatively, responsive to the execution of the plurality of first APIfunction calls, the intelligent optimization platform may function toinstantiate distinct clusters of tuning worker instances that eachoperate independently to optimize hyperparameters one of the pluralityof optimization work requests. That is, results of each of the distinctclusters of tuning worker instances may not interfere or influenceresults or generated values of other distinct clusters of tuning workerinstances.

Additionally, after the intelligent optimization platform has initiatedand/or completed work on the submitted optimization work request, S220,which includes implementing a second API function to create and/oraccess suggestions from the intelligent optimization platform for valuesof the tunable hyperparameters of the optimization work request,functions to enable a user to obtain or access one or more generatedsuggestions via the intelligent API. That is, S220 may function toexecute the second API call function to identify a plurality of rawvalues for the hyperparameters of an optimization work request that weregenerated by the intelligent optimization platform. In such embodiments,S220 may function to identify and/or obtain the plurality of raw valuesfor the hyperparameters using metadata provided with the optimizationwork request to query a platform database of the intelligentoptimization platform.

Accordingly, using the second API function enables a user of theintelligent API to provide input details optionally with metadata thatmay trigger the intelligent API to create and/or pull suggestions datafrom a platform database of the intelligent optimization platform. Insuch embodiment, S220 may function to generate one or more suggestionscomprising proposed hyperparameter values selected from a plurality ofraw values for the hyperparameters of the external machine learningmodel. The intelligent API may function to return the hyperparametervalues within the accessed suggestions data in any suitable mannerincluding by generating a suggestions assignments that outputs thesuggested parameter values along with any associated categorical values(if any) and combined with user-specified metadata that describes usesof the parameter values and the categorical values (if any).

Additionally, or alternatively, S220 may function to execute the secondAPI call function to instantiate a surrogate machine learning model forthe external machine learning model of a subscriber to the services ofthe platform. In such embodiments, instantiating the surrogate machinelearning model includes estimating an algorithmic structure of thesubscriber machine learning model based on the hyperparameters of thetuning work request and the plurality of raw values for each of thehyperparameters of the subscriber machine learning model and using theestimation of the algorithmic structure of the subscriber machinelearning model to define and/or create the surrogate machine learningmodel. For instance, the estimation of the algorithmic may be used asinput into a selected optimization or tuning source of the intelligentoptimization platform to thereby generate the surrogate machine learningmodel. In such embodiments, S220 may function to use the surrogatemachine learning model to test a performance of the plurality of rawvalues of the hyperparameters based on input into a structure of thesurrogate model of the plurality of raw values for each of thehyperparameters of the external machine learning model.

In a preferred embodiment, the surrogate machine learning modelcomprises a hypothesis of or a proxy for the external machine learningmodel when a structure of the subscriber machine learning model is notknown to the intelligent optimization platform.

Additionally, or alternatively, based on results of testing theplurality of raw hyperparameter values with the surrogate model, S220may function to identify a subset of the plurality of raw values foreach of the hyperparameters for generating one or more suggestions.Accordingly, the one or more suggestions may include proposedhyperparameter values identified from the subset of the plurality of rawvalues. In some embodiments, S220 may function to identify the subset ofthe plurality of raw values for each of the hyperparameters includesidentifying distinct raw values from the plurality of raw values thatsatisfy or exceed a performance threshold. The performance threshold maybe a predetermined or static value, in some embodiments. In otherembodiments, the performance threshold may be a dynamic value thatchanges according to a number of raw value for the hyperparametersgenerated by the intelligent optimization platform. For instance, if anumber of raw hyperparameter values generated by the intelligentoptimization platform increases, a minimum value of the performancethreshold may dynamically increase. The converse may also be true, insome embodiments.

Additionally, or alternatively, in some embodiments, S220 may functionto generate a surrogate machine learning model based on a detection orexecution of an intervening API call function. That is, in suchembodiments, S220 may function to execute an intervening API callfunction that operates to generate the one or more suggestions ahead ofan estimated or a scheduled completion of an optimization work request.Accordingly, in response to executing the intervening API call function,S220 may function to instantiate a surrogate machine learning model inlieu of the external machine learning model and test a performance ofthe plurality of raw values for each of the hyperparameters wheninstalled in the surrogate machine learning model.

Additionally, or alternatively, S220 may function to configure theintelligent API to enable parallel creation and/or access of suggestionsin circumstances when a user has multiple pending optimization workrequests being processed or that have been completely processed by theintelligent optimization platform. In such embodiments, the intelligentAPI may enable a user to input details for multiple optimization workrequest and correspondingly, return via intelligent API output, multiplesuggestions that include associated multiple assignments (e.g.,suggested parameter values).

Additionally, or optionally, S225, which includes evaluating theassignment values of a suggestion returned via the intelligent API,enables the user and/or user system to evaluate the suggestedhyperparameter values in a real-world (external) model. Preferably, theuser and/or user system may function to collect observations dataincluding performance values, such as resulting values for observedmetrics (e.g., accuracy, efficiency/speed, etc.) of the model.

S230, which includes implementing a third API function to create anobservation object via the intelligent API, functions to enable acreation of an observation object that includes input data relating toor including collected observations data. The observations data mayinclude any measured values of the metrics of the user's model togetherwith additional data or values that may have influenced a performance ofthe user's model.

The third API function of the intelligent API may additionally enablethe user to identify to provide an indication or data describing thesuggested parameter values or a suggestions ID of a suggestion which wasused in generating the real-world observations. In this way, a systemimplementing the method 200 may associate the collected observationsdata with the corresponding suggestions at the platform data of theintelligent optimization engine.

S240, which includes implementing an optimization feedback loop based onreceipt of observations data via the intelligent API, functions toenable an optimization feedback loop for optimizing parameter valuesbased on actual or real-world observed data values. The optimizationfeedback loop preferably includes looping S220-S230 until a desiredoptimization criteria set by a user is satisfied or met. Theobservations data, as referred to herein, may generally refer toobserved data that is collected during a trial implementations with thesuggestion parameters values being used as input into an external (orreal-world) model.

Accordingly, once the generated suggestions are made available to auser's system, real-world implementation of the suggestions parametervalues may be performed using an external model and the observed resultsof a measured metric of the model (e.g., observations data) may bereported back via the intelligent API. The report of the observationsdata may trigger or continue an optimization feedback loop that involvesthe receipt of observations data, the generation of new suggestionsbased at least in part on the observations data and making the updatedsuggestions available to the user's system via the intelligent API. Theoptimization feedback loop may persist until a desired value of themeasured metric of the external model is achieved.

In some embodiments, the optimization feedback loop of S240 may beautomatically stopped based on one or more predetermined stoppingcriteria. The one or more stopping criteria may relate to any one ormore of a plurality of user-designated triggers or implied (e.g., basedon inputs defined via the first API function for the optimization workrequest) triggers that function to indicate to the intelligentoptimization platform executing the optimization work request that acease of optimization trials are desired or required.

In some embodiments, the one or more predetermined automatic stoppingcriteria includes one or more predetermined thresholds that may beapplied to the one or more suggestions generated for the parameters ofan optimization work request. In such embodiments, if the parameters ofthe optimization work request meet or satisfy the one or morepredetermined threshold, this may trigger a stop optimization request,which instructs a system and/or intelligent optimization platform tocease in the execution of the subject optimization work request. Forinstance, if a metric of goodness or fit of a generated suggestion for amodel meets or exceeds a pre-set performance threshold, this may triggera stop optimization request.

The system and methods of the preferred embodiment and variationsthereof can be embodied and/or implemented at least in part as a machineconfigured to receive a computer-readable medium storingcomputer-readable instructions. The instructions are preferably executedby computer-executable components preferably integrated with the systemand one or more portions of the processors and/or the controllers. Thecomputer-readable medium can be stored on any suitable computer-readablemedia such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD orDVD), hard drives, floppy drives, or any suitable device. Thecomputer-executable component is preferably a general or applicationspecific processor, but any suitable dedicated hardware orhardware/firmware combination device can alternatively or additionallyexecute the instructions.

Although omitted for conciseness, the preferred embodiments includeevery combination and permutation of the implementations of the systemsand methods described herein.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

What is claimed is:
 1. A system implementing an application programminginterface of a tuning service for tuning one or more hyperparameters ofa model, the system comprising: an intelligent application programminginterface (API) that is in operable communication with API tuningsources of a tuning service and that executes: an API request thatinitiates a tuning of hyperparameters of a model of a subscriber to thetuning service, wherein: the API request creates a tuning work requestthat sets a plurality of tuning parameters for tuning thehyperparameters of the model, and each API tuning source of theplurality of distinct API tuning sources executes the tuning workrequest using a distinct set of tuning worker instances that executedistinct tuning tasks for tuning the hyperparameters of the model basedon the plurality of tuning parameters; an API call that causes the APIto: identify a plurality of raw values for the hyperparameters of themodel generated by each of a plurality of distinct API tuning sources;and return a response to the API call that includes one or more distinctsuggestions based on the plurality of raw values for the hyperparametersof the model, wherein the one or more distinct suggestions includesproposed hyperparameter values for the model selected from the pluralityof raw values for the hyperparameters of the model.
 2. The systemaccording to claim 1, wherein: the API request creates a tuning workrequest that sets a plurality of tuning parameters for tuning thehyperparameters of the model, and each API tuning source of theplurality of distinct API tuning sources executes on the tuning workrequest independently of and in parallel with other API tuning sourcesof the plurality of distinct API tuning sources.
 3. The system accordingto claim 1, wherein the intelligent API further includes: an APIobservations function that returns to the tuning service performancemetrics of the model based on the proposed hyperparameter values.
 4. Thesystem according to claim 1, wherein the hyperparameters compriseparameters of the model that have values set before a training or alearning process applied to the model and are not derived during thetraining or the learning process.
 5. The system according to claim 1,wherein: the API request creates a tuning work request that sets aplurality of tuning parameters for tuning the hyperparameters of themodel, and the tuning service executing the tuning work requestincludes: instantiating a surrogate model for the model of thesubscriber; and using the surrogate model to test a performance of theplurality of raw values of the hyperparameters based on an input intothe surrogate model of the plurality of raw hyperparameter values of themodel.
 6. The system according to claim 5, wherein the surrogate modelcomprises a hypothesis of or a proxy for the model when a structure ofthe model is not known to the tuning service.
 7. The system according toclaim 5, wherein instantiating the surrogate model includes estimatingan algorithmic structure of the surrogate model based on thehyperparameters of the tuning work request and the plurality of rawvalues for each of the hyperparameters of the model.
 8. The systemaccording to claim 5, wherein instantiating the surrogate model furtherincludes using an estimation of the algorithm structure of the model todefine and/or create the surrogate model.
 9. The system according toclaim 5, wherein the tuning service executing the tuning work request ofthe API request includes: identifying a subset of the plurality of rawvalues for each of the hyperparameters based on results of the testingwith the surrogate model, wherein the proposed hyperparameter values ofthe distinct suggestions for the plurality of distinct API tuningsources comprise the subset of the plurality of raw values.
 10. Thesystem according to claim 1, wherein the tuning service includes: acluster of distinct API tuning sources that operate to generate theplurality of raw values for the hyperparameters for the model, whereineach of the plurality of distinct tuning worker instances selectivelyoperates one distinct API tuning source of the cluster of distinct APItuning sources.
 11. The system according to claim 1, wherein theintelligent API further includes: an API intervening call that generatesthe one or more distinct suggestions ahead of an estimated or ascheduled completion of the tuning work request, wherein in response toimplementing the API intervening call, instantiating a surrogate modelin lieu of the model; and testing a performance of the surrogate modelbased on input of the plurality of raw values for the hyperparameters ofthe model.
 12. The system according to claim 11, wherein in response tothe API intervening call, automatically selecting by one of a pluralityof distinct tuning worker instances one of the plurality of distinct APItuning sources of the tuning service to generate a surrogate model fortesting a performance of the plurality of raw values for thehyperparameters of the model.
 13. The system according to claim 1,wherein implementing the API request call sets a mode of operation ofthe tuning service that is selected from a plurality of distinct modesof operation based on at least one tuning parameter provided with atuning work request.
 14. The system according to claim 1, whereinimplementing the API request sets one or more switching conditions that,when satisfied, switches a mode of operation of the tuning service froma first mode of operation to a second mode of operation, wherein thetuning service optimizes the hyperparameters of the model differently inthe second mode of operation than in the first mode of operation.
 15. Amethod implementing an application programming interface of a tuningservice for tuning one or more hyperparameters of a model, the methodcomprising: implementing an intelligent application programminginterface (API) that is in operable communication with API tuningsources of a tuning service, wherein the implementing includes:initiating a tuning of hyperparameters of a model of a subscriber to thetuning service based on an API request, wherein: the API request createsa tuning work request that sets a plurality of tuning parameters fortuning the hyperparameters of the model, and each API tuning source of aplurality of distinct API tuning sources executes the tuning workrequest using a distinct set of tuning worker instances that executedistinct tuning tasks for tuning the hyperparameters of the model basedon the plurality of tuning parameters; identifying a plurality of rawvalues for the hyperparameters of the model generated by the pluralityof distinct API tuning sources based on an API call; and returning aresponse to the API call that includes one or more distinct suggestionsbased on the plurality of raw values for the hyperparameters of themodel, wherein the one or more distinct suggestions includes proposedhyperparameter values that are a subset of the plurality of raw valuesfor the hyperparameters of the model.
 16. The method according to claim15, wherein the API request creates a tuning work request that sets aplurality of tuning parameters for tuning the hyperparameters of themodel, and each API tuning source of the plurality of distinct APItuning sources executes on the tuning work request independently of andin parallel with other API tuning sources of the plurality of distinctAPI tuning sources.
 17. The method according to claim 1, wherein theintelligent API further includes: returning to the tuning serviceperformance metrics of the model based on the proposed hyperparametervalues based on an API observations function.
 18. A system implementingan application programming interface of a tuning service for tuning oneor more hyperparameters of a model, the system comprising: anintelligent application programming interface (API) that is in operablecommunication with API tuning sources of a tuning service and thatexecutes: an API request that initiates a tuning of hyperparameters of amodel of a subscriber to the tuning service, wherein: the API requestcreates a tuning work request that sets a plurality of tuning parametersfor tuning the hyperparameters of the model, and the tuning serviceexecuting the tuning work request includes: instantiating a surrogatemodel for the model of the subscriber; and using the surrogate model totest a performance of a plurality of raw values of the hyperparametersbased on an input into the surrogate model of the plurality of rawhyperparameter values of the model; an API call that causes the API to:identify the plurality of raw values for the hyperparameters of themodel generated by each of a plurality of distinct API tuning sources;and return a response to the API call that includes one or more distinctsuggestions based on the plurality of raw values for the hyperparametersof the model, wherein the one or more distinct suggestions includesproposed hyperparameter values for the model selected from the pluralityof raw values for the hyperparameters of the model.
 19. The systemaccording to claim 18, wherein: each API tuning source of the pluralityof distinct API tuning sources executes the tuning work request using adistinct set of tuning worker instances that execute distinct tuningtasks for tuning the hyperparameters of the model based on the pluralityof tuning parameters.